A SAS / AF Application for Parallel Extraction , Transformation , and Scoring of a Very Large Database

This paper describes a SAS/AF application to extract large volumes of data in parallel from a multiple-terabyte RDBMS and directly populate a parallel SAS data mart. During population, the application allows the user to perform CPUintensive data transformation/normalization operations in parallel. This application also allows models generated by Enterprise Miner software to be deployed in parallel to score the entire data mart, or subsets of it. Parallel execution is achieved using Torrent Systems’ Orchestrate application development and runtime environment, which allows the application to • extract data in parallel from a parallel RDBMS • load the results of SAS programs back into the database in parallel • process parallel data streams with parallel instances of a SAS DATA or PROC step for much higher throughput • store large data sets in parallel, providing faster access and eliminating storage restrictions • stream data between SAS steps without having to write intermediate results to disk. The performance benefits of executing SAS extracts and other processes in parallel are well documented. In both production and test environments, parallel processing has allowed SAS applications to process larger workloads more quicklytypically improving performance by a factor equal to the number of processors used. These applications typically show near linear scalability. (An example of linear scalability is where a12-processor system provides 12 times the performance of a single processor.) These results are documented in the IBM Whitepaper, “Achieving Scalable Performance for Large SAS Applications and Database Extracts.”